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ABSTRACT 



An important problem facing institutions of higher education 
is the number of students reporting that they are not adequately prepared for 
the difficulty of college- level courses. To meet this problem, a computerized 
adaptive testing package was developed that permitted remote placement 
testing of high school students via the World Wide Web. The rationale for 
developing the project, how the programming was formulated, implementation 
and security issues, and preliminary data on student performance/reactions to 
the testing are presented in this report. The sample for the study consisted 
of 223 high school students from a suburban school district. Seventy-two 
percent of the students were juniors- -grade point averages ranged from 1.5 to 
4.0 (on a four-point scale). Three placement tests (a holistically scored 
writing sample, a reading test, and a math placement exam) were adapted for 
administration over an Internet link. Results show that student performance 
and perceptions of the math and writing tests were generally positive. Plans 
are underway to use a computerized writing evaluation program to assess 
written essays and to project what a student's performance might be one or 
two years into the future. (RJM) 
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Abstract 

A computerized adaptive testing package was developed that 
permitted remote placement testing of high school students via the World 
Wide Web. The project was designed to alert secondary students to the 
rigors of college work, and to let them explore their post -secondary 
schooling options. This paper describes the rationale for developing 
the project, how the programming was formulated, implementation and 
security issues, and presents some preliminary data on student 
performance/reactions to the testing. 
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Computerized Adaptive Testing through the World Wide Web 
Introduction 

An important problem facing institutions of higher education is to 
alert secondary students to the rigors of post -secondary work. New 
incoming students often report that they are not adequately prepared for 
the difficulty of college-level courses. Moreover, remediation rates at 
some post -secondary institutions can run as high as 90%, especially in 
states that lack a comprehensive community college system ('Study 
Finds,' 1996; U.S. Department of Education, 1995). This lack of 
preparation can result in lower retention rates, higher levels of 
dissatisfaction, and uncompleted student goals. While high school 
counselors spend a good deal of their time preaching the benefits of 
taking challenging coursework, these voices of reason are not always 
heard . 

There are a plethora of factors that can impact on student 
preparedness and success for college, but predominant factors emerge 
from the literature: family support and opportunities for feedback. The 
family support literature is best summarized in Christenson, Rounds, and 
Gorney (1992) who found a strong positive relationship between student 
achievement and parent involvement in education. Based on a synthesis 
of 153 research findings, five components of this factor were 
identified. Each of these is summarized briefly in the sections below. 

Parent Expectation s and Attributions . Though there is concern about the 
use of different methods to define and measure key concepts, studies 
evaluating parents' expectations, future aspirations, or current 



CAT through the WWW 4 



expectations for children's academic performance, a general conclusion 
that can be drawn is that academic achievement is consistently 
correlated with high parent expectations and attributions. Some studies 
examined children's perceptions of parents' expectations and others 
measured expectations more directly by questioning parents. Whether 
parents' expectations and attributions initially affect children's 
achievement or vice versa is not known. Effort attributions (i.e., 'you 
did well because you tried hard and practiced a lot') was strongly 
related to positive achievement outcomes. Realistic, high parent 
expectations for children's school performance are associated with 
positive academic performance regardless if the children's perceptions 
or actual parent expectations are investigated. 

StFUetuye for Learning . Most studies investigated elementary students 

within the structure of the home environment and how the environment can 
be manipulated to encourage and support academic learning. Those 
students assigned homework achieved higher grades than those not 
assigned such work. The beneficial effects are especially clear in 
middle and high school. Graded homework has a stronger impact than work 
unevaluated. Feedback and reinforcement produce higher student 

achievement . Parents can be helpful in the home process by providing 
(a) appropriate space to complete homework, (b) an established schedule, 
(c) adequate lighting and essential materials, (d) teacher support by 
providing rewards, checking homework, and stressing the value of school. 

Home — Affective Environment . Home affective environment is defined as 

the emotional environment in the home, and focuses on the relationship 
between parents and their children. Most studies concerned elementary 
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students . 


A positive 


affective 


relationship between 


parents 


and 


children 


increases the 


likelihood 


that the child will 


initiate 


and 



persist in challenging intellectual tasks and be socially competent. A 
positive parent-child relationship is related to academic success. 

Discipline. Discipline refers to parenting methods used to control 
children's behavior. Most studies have investigated the effects of 
discipline on high school students, although some studies included 
preschoolers. One study found that shared responsibility (between 
parent and adolescent) for decisions was associated with higher grades. 
Students who make decisions on their own tended to have lower grades. 
In situations where parents controlled all decisions, no association 
with grades was found. Another study found that parent involvement had 
a consistent, positive effect on students' grades, but no meaningful 
effect on students' standardized achievement scores (Keith, 1991). In 
one study of high and underachieving high school boys, the high 
achievers described their parents as being less restrictive in 
comparison to the underachieving group. High achievers reported 
receiving more encouragement from their parents than the underachievers 
(Keith, 1991). 

Parent Involvement . Parent involvement is broadly defined to 
include various activities that allow parents to participate in the 
educational process at home. The effect of parent involvement has been 
examined almost entirely at the elementary school level. Epstein (1987) 
suggests five types of involvement: (a) basic obligations of families to 
build positive home conditions that support learning, (b) school -home 
communications about school programs and children's progress (e.g.. 
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notices, report cards, conferences, phone calls), (c) involvement at 
school (e.g., attend school functions, volunteer work), (d) involvement 
in learning activities at home (e.g., monitor, discuss, and help with 
homework), and (e) involvement in decision-making (e.g., participation 
in PTA) . Researchers who examined components of effective schools often 
found high levels of parent involvement. Achievement gains have been 
greatest when parents are involved at all levels of school life. In 
studies that specifically examined the effects of involvement increased 
parent involvement is positively correlated with increased student 
achievement . 

Feedback opportunities manifest themselves in a variety of ways. 
One exemplary project was administered by the California Postsecondary 
Education Commission (California Postsecondary Education Commission, 
1996) . The purposes of the project were to: (a) improve student 
learning and academic performance in California public schools, (b) 
reduce the need for California public universities to offer remedial 
instruction, and (c) promote equality and merit in the college 
admissions process. Several programs were supported through this 
effort, but they all had six common characteristics: 

• The purpose of the program was to increase the number of 
students who pursue higher educational opportunities from 
backgrounds and communities that have historically demonstrated 
low eligibility and college-going rates. 

• The common strategy to accomplish this programmatic goal was the 
development of a cooperative relationship between public schools 
and higher educational institutions. 
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• While the programs were locally- or regionally-based, they were 
administered at the state level through a central office. 

• The programs tended to serve students directly rather than 
concentrating on improving curriculum or instruction. 

• The programs focused on providing services to students from 
groups who have documented low eligibility and college-going 
rates . 

• The programs were centered on the transition between high school 
and college as contrasted to the flow within higher education. 

One effort supported by the California project was the Advancement 

Via Individual Determination (AVID) , a direct services program that 
involved 19,500 students. Strategies that the program used included: 
(a) preparation for college admissions and placement tests, (b) academic 
support in rigorous curriculum, (c) advisement and career preparation, 
(d) parent education, (e) instruction in writing and inquiry, and (f) 
tutors in academic courses. This was an impressive intervention in that 
91 percent of the AVID graduates in 1995 completed a University 
preparatory curriculum in contrast to the statewide rate in 1994 which 
was only 32.2 percent. 

One of the key components of a few California programs centered 
around the use of placement testing to prepare entering students for the 
rigors of college work. If a student takes a placement test in his/her 
junior year of high school, feedback can include a report on readiness 
for college coursework, and a revised high school curriculum if 
remediation is required. A barrier to administering placement tests of 
this kind has been the college/university's need to control security of 
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the test, yet make it available to those who would like to assess their 
skills . 

Most colleges and universities have developed or adopted placement 
tests for the purpose of directing students into appropriate math, 
reading, or English courses (Chronicle of Higher Education, 1996) . The 
rationale for placement testing is threefold. First, students who 
enroll in appropriate courses should have a more positive experience 
than those who enroll in courses that are either too difficult or too 
easy. They should be more satisfied with their college experience, and 
are therefore more likely to be retained. Second, because students are 
more likely to stay in classes that are appropriate to their ability 
level, departmental administrators can more carefully plan how to 
allocate faculty resources to class sections. Finally, the placement 
exams might serve as a basis for measuring the contributions of the 
University to the development of general education skills. 

One of the problems with the implementation of most placement 
tests is that they serve as a summative assessment of what the student 
has acquired in high school or through work. There is generally no 
opportunity to 'learn from the test' and alter study behavior or 
enrollment patterns. 

A factor that can facilitate the delivery of tests or distance 
learning activities is the presence of the World Wide Web (WWW) 
(Shiffler, 1996) . A web server can present hundreds of tests per hour 
with reasonable response time, and with some advanced planning, a modest 
amount of security (Shermis, 1997) . Moreover, this medium has eclipsed 
others in terms of its popularity and accessibility (Campbell, 1997) . 
WWW testing can be embedded within other educational activities or even 




9 



CAT through the WWW 9 



used as part of an 'edutainment' package. Figure 1 shows a screen of a 
current popular 'edutainment' package entitled, 'You Don't Know Jack' 
(Berkeley Systems, Inc., 1997). The premise of the program is similar 
to those portrayed on popular television quiz shows— one is given a 
question (i.e., usually obscure) and a compressed amount of time to 
answer the question. What is impressive about this package is its use 
of audio and visual special effects to entertain the contestant during 
the testing process, even over slow modem lines. The package can even 



monitor 


performance 


to the point 


where 


it can award a 


contestant 


monetary 


prizes . 


The program. 


like 


its commercial 


television 



counterpart, is liberally sprinkled throughout with commercials. 



Insert Figure 1 About Here 



Development Work 

The development of the testing package was conducted as part of a 
larger project involving an information kiosk created by a large 
midwestern university. The kiosk was sponsored by several enrollment 
management teams including representatives from the admissions office, 
financial aid, registrar, bursar, a freshmen/ sophomore student division 
and the testing center. The purpose of the team was to encourage 
prepared high school graduates to enroll in a post-secondary experience. 
The testing package was designed specifically to allow students, 
beginning in tenth grade, to practice their skills using actual college 
placement tests. Feedback was to be designed so that the students could 
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create a trajectory of what kind of entry-level college courses in which 
they could expect to enroll (e.g., remedial versus college level 
courses) . 

An urban middle-class high school cooperated in the development of 
the kiosk software. Representatives from the high school included math 
and English teachers, counselors, and senior-level administrators. The 
first step was to obtain consensus on the functions that the kiosk would 
maintain and to discuss the accessibility of the software. For example, 
a number of high school representatives wanted the software to be 
available at a local library or from a student's home. It was decided 
that some functions (e.g., financial aid searches) could be accomplished 
from the home, but that other functions (e.g., testing) had to be 
performed on-site to allay considerations of security. Furthermore, 
minimum hardware /software configurations were specified. For example, 
the local library maintained only a text -based browser because they 
feared that some customers would use the internet to search for 
inappropriate material (e.g., pornography). However, the text-based 
browser did not support the graphics that are required for some graphic- 
based problems (e.g., geometry) in the test item bank. 

Next, the teams from the University and high school went through a 
storyboarding process. Mock-up screens were generated by the 
development group for the kiosk information functions and presented to 
the teams. The teams essentially acted as a focus group for how they'd 
like to see the information presented. A number of problems were 
anticiapted in this way. For instance, the normal university web 
screens for the offices of the bursar, registrar, financial aid, and 
admissions were seen as too complex for an "outsider" to navigate. 
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Consequently simplified screens were created that had functionality of 
interest only to high school students. Later the reactions of the 
experts were confirmed with a small number of students. 

Once the storyboards were completed, the coding was done in both 
Java and HTML. With regard to the programming for placement testing, a 
number of logistics were confronted. Test registration, for example, 
required the creation of several common gateway interfaces (cgi) to 
accommodate test security. At the first step, a counselor or teacher 
registers a student to take a test or a battery of tests. This process 
results in the issuance of a password that is good for two weeks. The 
student is given the password, but must take the test in a proctored 
area. Since the cgi checks for IP numbers, an attempt to take the test 
in an unproctored area will result in an error message to the student. 
We are currently working on an inexpensive means to remotely monitor 
test taking behavior to discourse cheating or other divergences from 
standard test-taking conditions. 

Next, the web screens were pre-tested at the university to 
eliminate potential errors and 'bugs*. It was then piloted in a trial 
run at the high school. One problem encountered at the high school had 
to do with a relatively sophisticated 'firewall' that they had recently 
installed. A 'firewall' is a computer protocol that protects a network 
by isolating it from outside connections. The piloting process required 
programming modifications that adapted the system for this new 
configuration. Once the programming modifications were complete, the 
system was released for general use. One planned feature was not 
included in the first release of the web-based software. Originally we 
had desired that college counselors might be available to consult with 
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the high school students through a video and audio hookup. This 
feature, while important, was delayed until the second year of the 
project . 



Method 

Participants . The sample for the current study consisted of a total of 
223 high school students from a suburban school district in a large 
midwestern city. The sample was comprised of 51% females and 49% males. 
No ethnicity information was collected, though the district is 
predominantly white. Twenty-six percent of the sample were high school 
seniors, seventy-two percent came from the junior class, and only two 
percent were sophomores. GPA ranged from 1.5 to 4.0 (on a 4-point 
scale), with a median GPA of 3.0. 

Instruments . Three placement tests were adapted for administration over 
an internet link between the two institutions. The first test ps a 
holistically scored writing sample based on a rotating set of prompts 
developed by faculty from the English department. These prompts are 
designed to focus students' discourse on a topic that would be in the 
realm of their normal experience. Students are given an hour to 
brainstorm, organize, and write an essay that is usually about 500 words 
in length. At least two instructors from the department evaluate the 
sample, and assign it a rating. The test was modified so that the high 
school students could type their responses into a web form document and 
submit it over the Internet directly to the department for evaluation. 
Feedback is provided in the form of an e-mail message to the student and 
counselor in about three days. Figure 2 illustrates the essay entry 



CAT through the WWW 13 



screen. The completed information is submitted via the web browser to a 
database and then is evaluated on a web screen by evaluators from the 
Department of English. 



Insert Figure 2 About Here 



The second test was the Indiana-Purdue Reading Test, an objective 
reading assessment consisting of five parts — reading speed, 
comprehension, . and three different types of vocabulary (Shermis, 
Wolting, & Lombard, 1996). The total reading score ranges from 0 to 
153, and represents a weighted raw score composite. The correlation 
between the Nelson-Denny , Form E (Brown, Bennett, & Hanna, 1981) and the 
new test is modest, but significant [£(348) = .37, p < .001]. Internal 
consistency is fair, ranging from £L - .58 to = .69. The results of 
separate factor analyses for each subscale showed evidence of construct 
validity and acceptable unidimensionality. Students who score less than 
80 are required to enroll in a developmental reading course. The test 
was modified for the Internet so that on the Internet it was 
administered through a common gateway interface (CGI) that called a web- 
compatible database. Feedback to the student and counselor is supplied 
immediately after the test is administered. Figure 3 provides an 
example of an item taken from this test. 



Insert Figure 3 About Here 
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The last test was a computerized adaptive math placement exam 
(Hsu Sc Shermis, 1989) . The testing mechanism utilized HyperCAT™, a 
computerized adaptive testing program that runs on either HyperCard (for 
Apple Macintosh) or WinPlus (for Windows /Windows 95) as a front-end 
processor. The testing algorithms were modified to run in a web 
environment. An adaptive test is one that conforms to the ability 
levels of the examinee. Computerized adaptive tests have several 
advantages over traditional paper-and-pencil tests. These include: 
shorter test administration times, better ability estimation, and 
perceptions by most examinees that the test is challenging, but not 
overwhelming. Moreover, the computerized format should result in better 
test security, ensuring that the tests are self-paced, and can be given 
at times convenient to the student. The adaptive testing package was 
accessed via the Web through a CGI and database that called the adaptive 
testing calculation routines. Feedback is provided as soon as the test 
is complete. Figure 4 highlights one item taken from the test. 



Insert Figure 4 About Here 



In addition to taking the exams, students are encouraged to browse 
the Web to obtain information about college admissions, financial aid, 
and registration. Figure 5 shows a sample of enrollment management 
options . 




15 



CAT through the WWW 15 



Insert Figure 5 About Here 



Procedure . Students took the tests in a variety of formats. Almost all 
of the assessments were given as part of an overall classroom experience 
provided by their teacher in a laboratory setting during the regular 
school day. Once enrolled, students had to register themselves on the 
appropriate web page to collect background information. The greeting 
screen for this web page in included in Figure 6. The computer then 
directed the student to the appropriate test or battery of tests. 



Insert Figure 6 About Here 



These World V/ide Web tests were implemented in an environment that 
posed only a few constraints. First, the tests had to be accessible 
from a PC or Macintosh computer running Netscape 3.01 or higher (with 
Java extension) . The computer had to be equipped with both a sound card 
and an inexpensive digital camera. Second, the connection speed was to 
be ISDN or higher. Third, the computer connections were to have static 
IP addresses or a known range of addresses if allocated dynamically. 
And finally, the space in which tests were to be implemented would be 
monitored. 

Student Survey . The placement testing exit survey was designed to 
solicit information about students' computing background, their 
perceptions of the testing experience, expertise and disposition of the 
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proctors, and some demographic characteristics. The information 
obtained from the survey is used internally to monitor proctor behavior, 
assess student satisfaction with the test experience, suggest areas for 
improvement, and specify possible technical innovations and 
recommendations that might improve the efficiency and effectiveness of 
the placement testing process. The survey also provides valuable 
information for external communication purposes with other service 
units/institutions by demonstrating how the placement testing process 
fits in with other university enrollment activities. 

Results 

Measures of Academic Performance . This section presents a brief 

summary of the descriptive statistics as indicators of academic 
performance of the high school students in mathematics and English. No 
information was available on reading performance . 

The distribution of math test scores, based on an N = 35 is 
presented in Figure 7. The mathematics test scores ranged from 6 to 26, 
with a math mean score of 14.4 and a standard deviation of 4.72. These 
scores compared favorably to a college sample using the same test (U = 
11.68; = 5.66) (Shermis, 1996). 

Figure 8 shows the mathematics placement distribution. The 
respective course placements are based on the cutoff scores for one 
university, but may not apply directly to other locales. For puposes of 
interpretation, Math M010 and Math 001 would be considered remedial. 
Math 111 would the first college-level course, and the entry labelled 
'See Advisor' would be beyond the entry-level. 
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Similarly, Figure 9 presents the placement distribution for 
English, based on a sample of 188 students. The English placement test 
scores ranged from 2 to 21, with a mean of 12.04 and a standard 
deviation of 3.33. Courses labelled EO10 and W001 are considered to be 
developmental. Courses labeled W131 or W140 are considered to be 
college-level courses. 



Insert Figure 7-8 About Here 



Student Satisfaction Survey . Satisfaction responses were obtained from 
60 of the 223 students who participated in the testing program for an 
overall response rate of 26.9%. Overall, the results of the present 
survey suggest that students' disposition towards the computerized 
placement testing program was fairly positive. For instance, 26.7% (or 
16 of the 60 students) had a favorable attitude toward the computerized 
testing process. In contrast, 25% (or 15 out of 60) of the respondents 
felt they could have done better if they had xxo!L taken the computerized 
placement tests. Approximatley 48% of the respondents gave a 'neutral ' 
response (on a five-point Likert scale) to the item: 'I would have done 
better if I had not taken the tests on the computer'. Note that a 
majority (55.7% ) of the students said they had very little or somewhat 
limited experience with computers; whereas 44.3% (or 27 out of 61 
students) reported having a great deal of computer experience. 

With respect to efficacy of the test directions, a majority (75% 
or 45 out of 60) of the respondents found the current test directions to 
be clear and easy to follow. In contrast, only 11.7% of the respondents 
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felt the test directions were unclear and rather difficult to follow. 
Furthermore, 95.1% of the students reported that use of computers in 
test-taking was very easy or not too bad. Only 3 out of the 61 students 
(4.9%) found the computerized testing process to be confusing. The 
results with respect to test anxiety were rather mixed: 37.7% of the 
respondents felt the testing experience made them less anxious about 
getting into college; 21.3% felt more anxious; and 41% of the students 
were 'undecided*. However, a majority of the students (61.7%) said the 
testing experience made them to 'realize the importance of the stress 
put on taking advanced courses to prepare for college'. 

With respect to computer knowledge of test proctor(s), a majority 
of the respondents (81.7%) had the perception that the proctor(s) had 
sufficient or extensive knowledge of computers. Fifteen percent of the 
respondents, however, found the proctor (s) to have inadequate computer 
knowledge. Perhaps this was due to some rather complex network problems 
that were encountered during the initial test sessions. 

Student perception regarding the accuracy of the placement tests 
was assessed too, and the appropriate results are presented in Figure 
10. For the present study, approximately 3 5% of the students (6 out of 
17) who took the mathematics placement test found the exam to be an 
accurate measure of math skills; whereas about 65% (11 out of 17 
respondents) felt that the math test was either too demanding or not a 
good measure of knowledge. With respect to the English placement test, 
approximately 62% (26 out of 42) of the respondents who took the exam 
felt it was an accurate measure of knowledge. On the contrary, a 
minority of the students (about 38%) found the English placement test to 
be too great of a demand or not a good measure of knowledge. Also, a 
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relatively small proportion of the respondents (31.1%) felt the English 
exam was a valuable learning experience. 



Insert Figure 10 About Here 



Di scu s s i o n 

This project illustrates a new, and perhaps innovative, way of 
applying internet technology to address the problem of under-prepared 
students who wish to pursue post -secondary education. One of the 
applications, computerized adaptive testing through the WWW, shows 
promise for allowing students to take the placement tests multiple times 
without compromising the test form. This project, and preliminary 
report, has shown the viability of using web-based technology for such 
an academic intervention. . 

While student performance and perceptions of the math and writing 
tests were generally positive, there are still a number of issues that 
have yet to be resolved. For example, while it is possible to make 
college placement recommendations based on a high school student's 
current level of performance, it is difficult to extrapolate that 
prediction one or two years into the future. That is, what exactly does 
one say to a high school sophomore who places into remedial college 
math? We wouldn't necessarily expect college-level performance, so what 
utility is there to provide feedback that confirms this? We are 
presently working to develop feedback that predicts where one is likely 
to stand in the future, given their current ability estimate and 
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curricular track. The IRT methodology used for the computerized 
adaptive tests should make the extrapolation process easier, though the 
predictions require empirical verification. 

Our efforts to generate appropriate feedback will extend to the 
written essay test as well. For example, we are currently exploring the 
possibility of integrating the testing system with the Essay Grader, a 
computerized writing evaluation page developed by Dr. Ellis Page of Duke 
University (Page & Peterson, 1995) . The Essay Grader, which has 
achieved remarkable indicators for reliability and validity (Page, 
Keith, Sc Lavoie, 1995), is normed on a secondary student population. If 
properly integrated with the web-based testing package, the Essay Grader 
could conceivably deliver normed and scored written essays within a 
matter of minutes. 
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